feat(chain): Improved `changes` RPC #2148

frol · 2020-02-18T03:20:15Z

Resolves #2034 and #2048

The changes RPC API was broken implementation-wise (#2048), and design-wise (#2034).

This version has meaningful API design for all the exposed data, and it is also tested better. This PR is massive since, initially, we missed quite a point of exposing deserialized internal data (like account info, access key used to be returned as borsh serialized blob, which is useless for the API user as they don't have easy access to the schema of those structures).

I will address #2129 in a separate PR.

API

EXPERIMENTAL_changes RPC endpoint accepts the following types of requests:

{
    "block_id": /* block hash | block height | omit it if you want to use `finality` */,
    "finality": /* "latest" | "final" | "near-final" | omit it if you want to se `block_id` */,
    "changes_type": /* "account_changes" | "single_access_key_changes" | "all_access_key_changes" | "code_changes" | "data_changes" */,
    /* extra fields based on `changes_type` */
}

the extra fields based on changes_type are the following:

"account_changes":
- account_id
"single_access_key_changes":
- account_id
- access_key_pk (public key)
"all_access_key_changes":
- account_id
"code_changes:
- account_id
"data_changes":
- account_id
- key_prefix_base64 (base64-encoded string of the key-prefix data saved from a smart contract)

Example of Account changes

When an account gets modified (created, updated, removed), we can query the block which contains the receipts for the transaction:

http post http://localhost:3030 jsonrpc=2.0 id=dontcare \
    method=EXPERIMENTAL_changes \
    'params:={
        "block_id": "DYHRzRbxUR1ANPdCQcgQE9g5zyYQoDZ1k8BJEQ3hDSgW"
        "changes_type": "account_changes",
        "account_id": "new-account"
    }'

we observe the following:

{
    "block_hash": "DYHRzRbxUR1ANPdCQcgQE9g5zyYQoDZ1k8BJEQ3hDSgW",
    "changes": [
        {
            "cause": {
                "type": "receipt_processing",
                "receipt_hash": "8rPN9tHY9MtT23TqBvvBGBQzwskA5fuZDjmiLpTobgRv"
            },
            "type": "account_update",
            "change": {
                "account_id": "new-account",
                "amount": "0",
                "locked": "0",
                "code_hash": "11111111111111111111111111111111",
                "storage_usage": 100,
                "storage_paid_at": 9
            }
        }
    ]
}

If the account was deleted:

{
    "block_hash": "DYHRzRbxUR1ANPdCQcgQE9g5zyYQoDZ1k8BJEQ3hDSgW",
    "changes": [
        {
            "cause": {
                "type": "receipt_processing",
                "receipt_hash": "8rPN9tHY9MtT23TqBvvBGBQzwskA5fuZDjmiLpTobgRv"
            },
            "type": "account_deletion",
            "change": {
                "account_id": "new-account"
            }
        }
    ]
}

Example of Data changes

When some key gets modified (created, updated, removed), we can query the block which contains the receipts for the transaction:

http post http://localhost:3030 jsonrpc=2.0 id=dontcare \
    method=EXPERIMENTAL_changes \
    'params:={
        "block_id": "DYHRzRbxUR1ANPdCQcgQE9g5zyYQoDZ1k8BJEQ3hDSgW"
        "changes_type": "data_changes",
        "account_id": "new-account",
        "key_prefix_base64": "bXlfa2V5"  /* base64-encoded "my_key" */
    }'

we observe the following (assume the key was modified twice in a single block, and then removed):

{
    "block_hash": "DYHRzRbxUR1ANPdCQcgQE9g5zyYQoDZ1k8BJEQ3hDSgW",
    "changes": [
        {
            "cause": {
                "type": "receipt_processing",
                "receipt_hash": "111N9tHY9MtT23TqBvvBGBQzwskA5fuZDjmiLpTob111",
            },
            "type": "data_update",
            "change": {
                "account_id": "new-account",
                "key_base64": "bXlfa2V5",  /* base64-encoded "my_key" */
                "value_base64":  "bXlfdmFsdWVfMQ==" /* base64-encoded "my_value_1" */,
            }
        },
        {
            "cause": {
                "type": "receipt_processing",
                "receipt_hash": "222N9tHY9MtT23TqBvvBGBQzwskA5fuZDjmiLpTob222"
            },
            "type": "data_update",
            "change": {
                "account_id": "new-account",
                "key_base64": "bXlfa2V5",  /* base64-encoded "my_key" */
                "value_base64": "bXlfdmFsdWVfMg==" /* base64-encoded "my_value_2" */
            }
        },
        {
            "cause": {
                "type": "receipt_processing",
                "receipt_hash": "333N9tHY9MtT23TqBvvBGBQzwskA5fuZDjmiLpTob333"
            },
            "type": "data_deletion",
            "change": {
                "account_id": "new-account",
                "key_base64": "bXlfa2V5"  /* base64-encoded "my_key" */
            }
        }
    ]
}

Test plan

I did not succeed in writing Rust tests as we used a mocked runtime there. Thus, I had extended end-to-end tests (pytest) with:

Test for account changes on account creation
Test for access key changes on account creation and access key removal
Test for code changes
Test for several transactions on the same block/chunk

gitpod-io · 2020-03-06T06:17:01Z

The `changes` RPC API was broken implementation-wise (#2048), and design-wise (#2034). This version has meaningful API design for all the exposed data, and it is also tested better. # Test plan I did not succeed in writing Rust tests as we used a mocked runtime there. Thus, I had extended end-to-end tests (pytest) with: * Test for account changes on account creation * Test for several transactions on the same block/chunk

frol

The implementation is finished. I will add a test for access key changes, but other than that, PR is ready for review.

frol · 2020-03-06T06:20:22Z

chain/jsonrpc/tests/rpc_transactions.rs

-                                            .map(|_| ()),
-                                    );
-                                }
+                actix::spawn(view_client.send(GetBlock::latest()).then(move |res| {


It is just a change in code formatting due to GetBlock::latest() change.

frol · 2020-03-06T06:24:12Z

chain/network/src/types.rs

@@ -1198,7 +1196,7 @@ pub enum NetworkViewClientMessages {
    /// Transaction status response
    TxStatusResponse(FinalExecutionOutcomeView),
    /// General query
-    Query { query_id: String, block_id: MaybeBlockId, request: QueryRequest, finality: Finality },
+    Query { query_id: String, block_checkpoint: BlockCheckpoint, request: QueryRequest },


It did not make sense for block_id and finality to co-exist. I had to make these changes in this PR as I was extending changes API (I should have created a separate PR, but it was hard to decouple the changes).

frol · 2020-03-06T06:24:31Z

core/primitives/Cargo.toml

@@ -6,11 +6,11 @@ edition = "2018"

 [dependencies]
 regex = "1"
-bincode = { version = "1.0", features = ["i128"] }


It is not used anymore.

frol · 2020-03-06T06:25:14Z

core/primitives/Cargo.toml

 bs58 = "0.3"
 base64 = "0.11"
 byteorder = "1.2"
 chrono = { version = "0.4.4", features = ["serde"] }
+derive_more = "0.99.3"


It helped to clean up some duplicate code around AsRef / Into / etc common implementations.

frol · 2020-03-06T06:34:56Z

core/primitives/src/serialize.rs

@@ -78,34 +43,6 @@ pub trait BaseDecode: for<'a> TryFrom<&'a [u8], Error = Box<dyn std::error::Erro
    }
 }

-macro_rules! pretty_bytes_serialization {


I decided to be more explicit and use _base64 suffix on the fields serialized with base64 encoding, which requires serde renaming anyway, so I rolled back this macro implementation and replaced it with appropriate field attributes.

frol · 2020-03-06T06:36:36Z

core/primitives/src/views.rs

@@ -1041,19 +1059,122 @@ pub struct GasPriceView {
    pub gas_price: Balance,
 }

-/// Different types of finality.
-#[derive(Serialize, Deserialize, BorshSerialize, BorshDeserialize, Clone, Debug, PartialEq, Eq)]
-pub enum Finality {


Moved to types.rs

frol · 2020-03-06T06:40:24Z

near/tests/stake_nodes.rs

-                        },
-                    ),
-                );
+                actix::spawn(test_nodes[0].view_client.send(GetBlock::latest()).then(move |res| {


Nothing except GetBlock::latest() has changed in this block.

frol · 2020-03-06T06:41:54Z

pytest/tests/sanity/rpc_key_value_changes.py

+expected_changes_response = {
+    "block_hash": state_changes_request["block_id"],
+    "changes": [
+        {
+            "cause": {
+                "type": "receipt_processing",
+                "receipt_hash": new_account_response['result']['receipts_outcome'][0]['id']
+            },
+            "type": "account_update",
+            "change": {
+                "account_id": new_account_id,
+                "amount": "0",
+                "locked": "0",
+                "code_hash": "11111111111111111111111111111111",
+                "storage_usage": 100,
+            }
+        }
+    ]
+}


Example of the RPC response for account changes.

frol · 2020-03-06T06:43:36Z

pytest/tests/sanity/rpc_key_value_changes.py

+expected_changes_response = {
+    "block_hash": state_changes_request["block_id"],
+    "changes": [
        {
-            "changes_type": "data_changes",
-            "account_id": tx_account_id,
-            "key_prefix": list(b"mykey")
+            'cause': {
+                'type': 'receipt_processing',
+            },
+            'type': 'data_update',
+            'change': {
+                'key_base64': base64.b64encode(b"my_key").decode('utf-8'),
+                'value_base64': base64.b64encode(b"my_value_1").decode('utf-8'),
+            }
+        },
+        {
+            'cause': {
+                'type': 'receipt_processing',
+                'receipt_hash': function_call_2_response['result']['receipts_outcome'][0]['id']
+            },
+            'type': 'data_update',
+            'change': {
+                'key_base64': base64.b64encode(b"my_key").decode('utf-8'),
+                'value_base64': base64.b64encode(b"my_value_2").decode('utf-8'),
+            }
        }
-    )
-    assert not deepdiff.DeepDiff(changes_from_another_node, changes), "query same changes gives different result"
+    ]


Example of changes RPC response when two transactions in a single block cause a change in the same key.

frol · 2020-03-06T06:46:09Z

core/primitives/src/utils.rs

-    key
-}
+#[derive(derive_more::AsRef, derive_more::Into)]
+struct KeyForColumnAccountId(Vec<u8>);


I had to refactor this file as I was adding a few more functions and already felt like loosing track of them, so I essentially grouped the functions into namespaces. /cc @mikhailOK it seems that you have been working on these before. Please, review the changes.

frol · 2020-03-06T11:05:27Z

chain/chain/src/store.rs

@@ -923,50 +923,175 @@ impl ChainStoreAccess for ChainStore {
        state_changes_request: &StateChangesRequest,
    ) -> Result<StateChanges, Error> {
        use near_primitives::utils;
+


FYI, the essential changes are in this method.

frol · 2020-03-06T11:08:44Z

chain/chain/src/store.rs

+                let mut changes = StateChanges::new();
+
+                for change in changes_iter {
+                    let (_, state_changes) = change?;
+                    changes.extend(state_changes.into_iter().map(|(cause, state_change)| {


I have tried to express it in less code (e.g. I wanted to use flat_map, but we want to short-circuit when the iterator fails), and I have also tried to generalize the code to avoid semi-duplication in the match branches, but it quickly becomes overcomplicated.

what is all the added code for?

We track all the changes to trie by simply saving all the key-values that go into it (the keys have their custom encoding schema indicating what type of data it is, and the values are some blobs [when smart-contract sets a key, we don't even know what format the value meant to have, thus data changes are simple -- we just return the key-value blobs, but when we store account/access-key/code updates, we internally borsh-serialize them and save into trie, so this code does deserialization from borsh into rust structs, so RPC can return those as JSON instead of some magical borsh-serialized blobs as it used to be before this PR]).

P.S. I will add this comment into the code. Good call.

bowenwang1996 · 2020-03-06T18:06:24Z

chain/chain/src/store.rs

+                let mut changes = StateChanges::new();
+
+                for change in changes_iter {
+                    let (_, state_changes) = change?;
+                    changes.extend(state_changes.into_iter().map(|(cause, state_change)| {


what is all the added code for?

chain/jsonrpc/src/lib.rs

core/primitives/src/types.rs

…48-query-account-changes

MaksymZavershynskyi

Please extract From trait implemenation from get_key_value_changes and consider renaming BlockCheckpoint to BlockIdOrFinality

chain/chain/src/store.rs

core/primitives/src/types.rs

MaksymZavershynskyi · 2020-03-09T15:38:07Z

Is this ready to be merged?

frol · 2020-03-09T15:58:32Z

I have addressed all the comments and finishing the test for Access Key changes, I will oush all of that in 3 hours, and it will be ready to be merged.

bowenwang1996 · 2020-03-09T17:49:08Z

looks like it needs @evgenykuzyakov's approval as well.

MaksymZavershynskyi · 2020-03-09T17:50:06Z

looks like it needs @evgenykuzyakov's approval as well.

I think it just needs the rebase

…48-query-account-changes

frol · 2020-03-10T02:23:19Z

I addressed all the comments. It is fine to be merged.

core/primitives/src/types.rs

…48-query-account-changes

frol · 2020-03-10T23:52:46Z

The documentation request is near/docs#173

@Kouprin

* Enable floats but prohibit some CPU architectures (#1941) (#2079) * Enable floats but prohibit some CPU architectures * Merge branch 'staging' into enable_floats * Merge refs/heads/staging into enable_floats * Avoid overflowing u32 during contract preparation (#1946) * Update runtime/near-vm-runner/src/runner.rs Co-Authored-By: Evgeny Kuzyakov <[email protected]> * Merge branch 'staging' into enable_floats * Nit Co-authored-by: Maksym Zavershynskyi <[email protected]> * Move sysinfo fix from staging (#2085) * Move sysinfo fix from staging * Introduce keccak256 and keccak512 native support (#2072) * Introduce keccak256 and keccak512 native support * modify genesis Co-authored-by: Bowen Wang <[email protected]> * Create code-of-conduct.md (#1932) * Create code-of-conduct.md * Move code of conduct to proper name Co-authored-by: Illia Polosukhin <[email protected]> * Ref #2067: querying contract state (#2075) * Fix #2067: querying contract state * Change format to {key, value, proof} for state view. Proofs are empty for now * Change state dump to binary format (#2086) Instead of dumping state in a json file which is slow, this PR changes it to dumping state in a binary format. Fixes #2070. Test plan --------- Manually test this with a local node to make sure that the state is preserved correctly after state dumping. * Add json state dump for debugging (#2120) * Add json state dump for debugging * state_dump.json * fix * docs: add links to Issues/Milestones (#2144) * fix(runtime): Fix empty method_names bug (#2188) Fixes: #2183 # Test plan: - Extracted the code into utils crate and added unit tests. * State dump split (#2126) - Revert binary state dump - #1922 for separating config and records - scripts/new-genesis-from-existing-state.sh that dump state, calculate new genesis hash, upload to s3 - testnet genesis records separate from near binary - download testnet genesis from s3 in python start_testnet - check genesis hash when run testnet Co-authored-by: Evgeny Kuzyakov <[email protected]> Co-authored-by: Bo Yao <[email protected]> Test Plans ------------- - it can `near init --chain-id=testnet --genesis-config near/res/testnet_genesis_config.json --genesis-records <records download from s3> --genesis-hash <expected-hash>` to initialize testnet config from external genesis records - When `near run`, with incorrect `~/.near/genesis_hash` it will panic - It can start testnet with updated start_testnet.py, which download genesis_records from s3 - After stop a node, we can call `state-viewer dump_genesis` to dump genesis_records, config, and genesis_hash. - If there was no state change happen since genesis block, load the dump genesis_records/config will give you a same genesis_hash, and dump again generate same genesis_records, config & genesis_hash. * Avoid panic while panicking because of actix isn't running (#2178) Some of our tests don't have actix running. @Kouprin found when assert in such tests failed it's not able to see which test fails, only `thread panicked while panicking`. So check actix is running and only shutdown it if so fix this. Test Plan --------- ``` #[test] fn test_assert() { init_stop_on_panic(); assert(false); } ``` should give assert fails instead of `thread panicked while panicking` * Fix epoch manager bug (#2193) * feat(runtime): Validate incoming receipts #2155 If an invalid chunk made it into block the invalid incoming receipts might be forwarded to different shards. Ideally the chunk should be challenged and the block is reverted, but before this happens we need to handle invalid receipts in the Runtime. Another possibility is to create invalid receipts in the state directly and then create a challenge on this invalid state. So any field in the state can potentially contain invalid value. So if an invalid receipt is present in the delayed receipts state, we consider it a StorageError. Fixes: #1850 # Test plan - Added 2 tests to cover handling of invalid receipts in the Runtime - Filed an issue to handle error in NightshadeRuntime #2152 * fix: Use stderr for logs generated with tracing. (#2198) * fix: Fixing nightly python tests (#2201) * `block_production.py` started failing because the blocks are now produced faster, and by the first time it checked the heights the heights exceeded 2, causing poorly stated assert to trigger * `staking*.py` were never adjusted for yocto-near * `state_sync*.py` both header sync and state sync were broken Header sync: we were cleaning up mapping from heights to headers during GC, which is necessary for header sync. State sync: we changed the state sync to always happen at the epoch boundary, but did it in a wrong way: the hash we stored locally was pre-pushing to the epoch boundary, and thus the node was rejecting any incoming state responces * `state_sync1.py` was also failing because it was relying on one out of two validating nodes being able to produce blocks, which it cannot, because of Doomslug * `skip_epoch.py` was expecting one validator to be producing blocks, which with doomslug requires such validator to have more than half the stake. Also was not updated for yocto-near * `one_val.py` similarly was expecting one validator to be producing blocks, and thus needed that validator stake bumped to worh with DS * `lightclnt.py` with doomslug blocks are produced faster, and by the time the test started a few blocks were already produced. Making the test expect such behavior * `block_sync.py` was failing because Doomslug requires `max_block_production_delay` to be at least 2x `min_block_production_delay`. Also disabling `network_stress`, because the current runner crashes trying to launch it and skips all the consecutive tests Test Plan --------- All the above python tests pass. * fix: Fix pytest after log to stderr (#2203) Similar to #1985 but now we changed back to log to stderr Test Plans ------------- Run pytest locally, now it's no longer have `AssertionError: node dirs: 0 num_nodes: 4 num_observers: 0` error * fix: fixing cross_shard_tx* tests and small fixes for some other nightly failures (#2204) There were several issues in the test infra: 1. The peer info in the client test infra was the largest height and the largest score ever observed. If a block with a higher score but lower height than the previous tip was created, it would report incorrect peer info, and peers would attempt header sync believing the peer has higher header head height, and such header sync would fail. 2. The tests tamper with the FG, and the last final block could be way more than 5 epochs in the past. That makes creating light client blocks potentially require blocks from 5 epoch lengths ago. I'm just making all nodes in cross_shard_tx archival. In practice if one epoch has been lasting for five epoch lengths, we have bigger problems. 3. We historically see cross_shard_tx tests fail with `InvalidBlockHeight` error when a block is more than epoch length ahead of the previous block. Since that check is a heuristic anyway, I'm doubling the distance, to reduce the flakiness of the test. Separately, increasing the timeouts for NFG tests, they take more than 15 minutes. Also bumping timeouts for the `test_all_chunks_accepted_1000*` tests, it's clear that they need at least 2000 / 4000 / 1000 seconds to complete, I set the timeouts to 3600 / 7200 / 1800 for some extra room. Also the one that requires 7200 (`*_slow`) seems to provide no value compared to the base test, and is the slowest test in our entire suite, so I completely disable it. Separately, fixing the issue with state sync tests, where the transition to state sync happens before the log tracker is initialized, and the check for the transition in the log later fails Slightly bumping block production time in `test_catchup_sanity_blocks_produced`, it works on local machine, but on the gcloud runner doesn't keep up. Test plan --------- All cross_shard_tx* tests passed at least three runs. If they are flaky, nightly will catch that. * feat(chain): Exposed genesis config + runtime config and genesis records via RPC Resolves #2007 and #2025 We needed to make Near config query-able through node RPC. Specifically, one of our clients wanted to know how many blocks remain until an account is going to be evicted due to rent. This information can be derived from the account balance and the config. In this commit, two new RPC endpoints are exposed: EXPERIMENTAL_genesis_config and EXPERIMENTAL_genesis_records. Learn more in PR #2109. # Test plan Added tests to query the endpoints with a happy path and also invalid parameters. * fix: Third wave of fixing nightly tests (#2207) Many timeout tweaks, and small typos. `cross_shard_tx_with_validator_rotation` with 150 block time has extremely long forks (doomslug is disabled), and takes lots of time per iteration. I split it into two: one with 150ms block time, but only 16 iterations, and one with 400ms block time, but all 64 iterations. Both locally take ~45 minutes, will see how long it takes on gcloud. * fix: Fixing old rust multi-node tests (#2210) Fixing the following issues: 1. `ThreadNode` was not properly setting its state on kill 2. `ThreadNode` doesn't properly free up its port, but instead of figuring out why, I just replaced it with `ProcessNode` in the tests that are affected 3. `test_4_20_kill1` wasn't accounting for fees. Disabling fees. 4. In the same test, the same chunk producer was always mapped to the same block producer, and thus killing the second node was making the 0-th chunk producer (who happens to be attached to the 2nd BP) to not be able to have their transactions included. Address it by having 17 seats. Also split the test in two, with one shard and with two shards 5. In multiple places we were using the wrong node to get the access key Test plan --------- Locally `test_4_20_kill1` and `test_*_multiple_nodes` pass, let's see how the next nightly looks * fix: Disabling some tests + small fixes (#2211) Disabling old tests that fail due to the runtime cache unti they are fixed; Changing the `cross_shard_tx_with_validator_rotation` slightly based on its performance on gcloud. (increasing the block prod time speeds up test, because it results in fewer forks, so the 150->200 change is to make more iterations fit. With 150ms it fits ~6 iterations into one hour) * Buildkite CI (#2217) The actual pipeline definition is saved on buildkite ui, this way, we got it shared between stable, beta and master. There'll be some master only builds (nightly release) Test Plans -------------- Buildkite CI should pass * fix(doctest): Refactor comment. (#2219) Remove doctest that was running incorrectly. * fix: Fixing stress.py (#2220) The test was not updated after some config parameters were renamed. Also because of #2195, tx status of lost transactions times out, so adding a workaround into the test for now Disabling the version of stress that messes up with network, because the nightly runner is currently not configured to support the utility I use to stop network between processes Couple other changes: 1. Made `node_restart` worker not restart the node if no blocks were produced in the meantime. It is needed because with only two nodes after a long restart doomslug can take a while to recover (it is equivalent to half the network shutting down and restarting after some delay). Block production worker will fail the test if the block production actually stalls 2. For the same reason increased the tolerated delays for block production 3. Limited how many txs are sent per iteration of tx worker, since due to #2195 it takes one second to query one transaction, and if the test finished in the middle of querying, the allowed one minute for workers to stop is not sufficient for the tx worker. 4. Also generally increasing the allowance from 1m to 2m at the end, since the tx worker at the end of the test might take some time before it even starts querying the transaction outcomes * Try setting up GitPod (#2190) * Try setting up GitPod * Try pre-building nearcore in Docker image * Remove rustup command (nightly should already be available) * Update location for nearcore prebuild * Fix the way cargo is executed in Dockerfile * Do cargo test as well when building docker image * fix(epoch manager): Fix fishermen unstake (#2212) * Fix fishermen unstake * fix kickout set * add test * fix coverage, badge, docker release on in master branch (#2218) Equivalent coverage and docker image release from gitlab ci. Except release is a docker image release and will also add s3 release in future, since gcloud storage requires google login, which is inconvenient for used in scripts Test Plans -------------- coverage only in master, beta, stable branch and docker release only in master branch (For test purpose also this cov-release branch but will be removed). badge updated. docker release in beta/stable branch will be added but not ready now as tests in beta/stable is more strict * feat(adversary): adv_disable_doomslug + tests update * fix(epoch manager): Fix validator kickout set (#2214) * Fix validator kickout set * fix stake change * fix(epoch_manager): Some fixes and comments (#2222) Some fixes, refactoring and comments. Test plan --------- Run existing tests Add more asserts * fix: Increasing tolerance in stress.py, and fixing nightly.txt (#2228) Further investigation of `stress.py` failures in nightly shows that all the workers appear to work as expected, but frequent restarts cause block production to be delayed more than the current tolerance. Locally spradically also more than 50% of transactions get lost if the node get restarted too frequently. Increasing both tolerances. Also making the prints unbuffered, otherwise several workers in the nightly runs don't flush their outputs. Finally, fixing a typo in the nightly.txt Test plan --------- Locally `stress.py` doesn't fail, so there's no easy way to test whether the new tolerances would be sufficient. It also doesn't completely fix all the known issues, there are some other failures in stress.py that I haven't gotten to yet * Add doc test run in CI (#2230) Add doc test runs in CI Test Plan ------------ Should see doc test log at begining of other tests * fix(runtime): Fix state change check during view call (#2229) Fixes #2226 The current behavior of a view call is to prohibit state changes. This change moves check from the end of the state viewer to VMLogic by prohibiting state changes functions during view calls. ## Test plan: - Fixed state change tests. Previously the test didn't verify error type. The error was `MethodNotFound` and also alice account was wrong. - Added unit tests for the new prohibited methods. * Bump Borsh and runtime version (#2232) * Bump Borsh version * Nit * Bump Borsh versions * Bump borsh more * Add binary release script (#2235) scripts for binary release Test Plan ------------ Download uploaded binary in a few popular linux vm and see if they works * Skip phantom in genesis serialization (#2238) * Improve 100 node test to not reserve static ip and firewall rule (#2241) Two people run 100 node together is often blocked by firewall rules limit and static ip address limit (The recent fail by @mfornet is this case). Made they do not reserve ip address and use organization global firewall rules would fix this. Test Plans -------------- The same setting has been proved to work in create devnet nodes. * feat(chain): Improved `changes` RPC (#2148) Resolves: #2034 and #2048 The changes RPC API was broken implementation-wise (#2048), and design-wise (#2034). This version has meaningful API design for all the exposed data, and it is also tested better. This PR is massive since, initially, we missed quite a point of exposing deserialized internal data (like account info, access key used to be returned as a Borsh-serialized blob, which is useless for the API user as they don't have easy access to the schema of those structures). ## Test plan I did not succeed in writing Rust tests as we used a mocked runtime there. Thus, I had extended end-to-end tests (pytest) with: * Test for account changes on account creation * Test for access key changes on account creation and access key removal * Test for code changes * Test for several transactions on the same block/chunk Co-authored-by: Anton Bukov <[email protected]> Co-authored-by: Maksym Zavershynskyi <[email protected]> Co-authored-by: Bowen Wang <[email protected]> Co-authored-by: AnaisUrlichs <[email protected]> Co-authored-by: Illia Polosukhin <[email protected]> Co-authored-by: Illia Polosukhin <[email protected]> Co-authored-by: Lazaridis <[email protected]> Co-authored-by: Evgeny Kuzyakov <[email protected]> Co-authored-by: mikhailOK <[email protected]> Co-authored-by: Marcelo Fornet <[email protected]> Co-authored-by: Alexander Skidanov <[email protected]> Co-authored-by: Bowen Wang <[email protected]> Co-authored-by: Vlad Frolov <[email protected]> Co-authored-by: Vladimir Grichina <[email protected]> Co-authored-by: nearprotocol-bulldozer[bot] <56702484+nearprotocol-bulldozer[bot]@users.noreply.github.com> Co-authored-by: Alex Kouprin <[email protected]>

This was referenced Feb 18, 2020

Failed to query account changes #2048

Closed

get_key_value_changes can panic when processing a query #2052

Closed

weekly-digest bot mentioned this pull request Feb 21, 2020

Weekly Digest (14 February, 2020 - 21 February, 2020) #2166

Closed

ailisp changed the base branch from staging to master February 25, 2020 01:07

weekly-digest bot mentioned this pull request Feb 28, 2020

Weekly Digest (21 February, 2020 - 28 February, 2020) #2205

Closed

frol force-pushed the fix/2048-query-account-changes branch from c3c8cd9 to 37d7113 Compare March 5, 2020 03:54

frol force-pushed the fix/2048-query-account-changes branch from 2f22a6a to f082d36 Compare March 6, 2020 06:31

frol commented Mar 6, 2020

View reviewed changes

frol marked this pull request as ready for review March 6, 2020 06:48

frol requested review from bowenwang1996, evgenykuzyakov, MaksymZavershynskyi and SkidanovAlex as code owners March 6, 2020 06:48

frol assigned MaksymZavershynskyi Mar 6, 2020

frol changed the title ~~fix: Failed to query account changes~~ feat(chain): Improved changes RPC Mar 6, 2020

frol commented Mar 6, 2020

View reviewed changes

bowenwang1996 reviewed Mar 6, 2020

View reviewed changes

frol added 2 commits March 6, 2020 17:43

addressed review comments

ffaa011

Merge branch 'master' of github.com:nearprotocol/nearcore into fix/20…

4fa0806

…48-query-account-changes

bowenwang1996 approved these changes Mar 6, 2020

View reviewed changes

frol mentioned this pull request Mar 6, 2020

Changes RPC should allow querying multiple accounts at once #2129

Closed

MaksymZavershynskyi approved these changes Mar 6, 2020

View reviewed changes

MaksymZavershynskyi assigned frol and unassigned MaksymZavershynskyi Mar 9, 2020

frol added 2 commits March 9, 2020 23:01

Merge branch 'master' of github.com:nearprotocol/nearcore into fix/20…

8bf6ca5

…48-query-account-changes

Addressed the review comments

772377d

frol assigned evgenykuzyakov and unassigned frol Mar 10, 2020

evgenykuzyakov requested changes Mar 10, 2020

View reviewed changes

core/primitives/src/types.rs Show resolved Hide resolved

frol added 3 commits March 10, 2020 16:48

Added missing account_id to all the changes API responses

925a5b3

Merge branch 'master' of github.com:nearprotocol/nearcore into fix/20…

18bc1ae

…48-query-account-changes

fixed the block_checkpoint -> block_id_or_finality renaming

60bd4ab

evgenykuzyakov approved these changes Mar 10, 2020

View reviewed changes

frol merged commit 6d07a2d into master Mar 10, 2020

frol mentioned this pull request Mar 10, 2020

[DOC] Document new changes RPC API near/docs#173

Closed

frol mentioned this pull request Mar 11, 2020

add RPC named params and finality flag. close #184 near/docs#252

Merged

frol mentioned this pull request Mar 17, 2020

expose balance-change events in a block #2036

Closed

frol deleted the fix/2048-query-account-changes branch March 20, 2020 14:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(chain): Improved `changes` RPC #2148

feat(chain): Improved `changes` RPC #2148

frol commented Feb 18, 2020 •

edited

Loading

gitpod-io bot commented Mar 6, 2020 •

edited

Loading

frol left a comment

frol Mar 6, 2020

frol Mar 6, 2020

frol Mar 6, 2020

frol Mar 6, 2020

frol Mar 6, 2020

frol Mar 6, 2020

frol Mar 6, 2020

frol Mar 6, 2020

frol Mar 6, 2020

frol Mar 6, 2020

frol Mar 6, 2020

frol Mar 6, 2020 •

edited

Loading

bowenwang1996 Mar 6, 2020

frol Mar 6, 2020

bowenwang1996 Mar 6, 2020

MaksymZavershynskyi left a comment

MaksymZavershynskyi commented Mar 9, 2020

frol commented Mar 9, 2020

bowenwang1996 commented Mar 9, 2020

MaksymZavershynskyi commented Mar 9, 2020

frol commented Mar 10, 2020

frol commented Mar 10, 2020

feat(chain): Improved changes RPC #2148

feat(chain): Improved changes RPC #2148

Conversation

frol commented Feb 18, 2020 • edited Loading

API

Example of Account changes

Example of Data changes

Test plan

gitpod-io bot commented Mar 6, 2020 • edited Loading

frol left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frol Mar 6, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaksymZavershynskyi left a comment

Choose a reason for hiding this comment

MaksymZavershynskyi commented Mar 9, 2020

frol commented Mar 9, 2020

bowenwang1996 commented Mar 9, 2020

MaksymZavershynskyi commented Mar 9, 2020

frol commented Mar 10, 2020

frol commented Mar 10, 2020

feat(chain): Improved `changes` RPC #2148

feat(chain): Improved `changes` RPC #2148

frol commented Feb 18, 2020 •

edited

Loading

gitpod-io bot commented Mar 6, 2020 •

edited

Loading

frol Mar 6, 2020 •

edited

Loading